So those of us working in the application security space have been watching with trepidation as the use of package managers and minimalist languages have cause exponential increases in application dependencies with not more than a little bit of fear. I remember some years ago when I was still in development, certain teams at a previous employer were moving to NodeJS for new projects more than a few engineers were concerned with the hundreds of dependencies they were including in their projects. It was really quite plain that there would be no way to review all the code being pulled in so the attitude we took was one of cross your fingers and hope for the best. Not exactly a good engineering technique.

The real wake up for me was in the following years when I transitioned to application security and became exposed to the vast swaths of vulnerabilities being publicly acknowledged by major vendors as that became a big thing. Just the sheer number of hard coded back doors and easily bypassed authentication systems from major vendors was crazy. It was mind boggling how much trust in large enterprise vendors had been misplaced. It was obvious back then that open source was generally better at preventing security problems just by being scrutable. And then the open source dependencies started having problems too. Not as bad as some large enterprise vendors, but still incidents were slipping through.

Now we seem to be at the beginning of an awakening of the massive weaknesses in IT supply chains. Obviously some of the recent attacks, like the one that hit SolarWinds, are very sophisticated and outside the realm of your average internet criminal organization. But with all the coverage and the release of techniques and malware source we can be sure there will be copy cats of lesser sophistication.

Sure there are some security systems an enterprise can implement to help. Software Composition Analysis tools can make you aware of your dependency scope, and let you act on known problems. Some repository management software will help you confirm things like hashes and let you slow down intake of new dependencies. But none of it would really help you against something targeted and stealthy like Sunspot.

That’s not to say nothing can be done to stop such an attack, it’s just what it would take is often well beyond what private enterprises and security vendors currently do. That’s not to mention that most engineers and managers in IT barely understand what the supply chain is, and often confuse it with the software development lifecycle which makes any defense placed too far down the line to be effective. So before I talk about speculative mitigations, let me define some logistics terms as I see them in software development:

"Upstream Supply Chain": Data, libraries and software that are brought into an enterprise to make software and services.

To understand how to defend this, it needs to be understood that this stops at some point where we need to enforce security before bad things can happen.

"Manufacturing Process": The process of creating code to create additional software or services, to support or as, the business practice. This manufacturing process is often referred to as the software development lifecycle (SDLC).

This is the development, manipulation or redistribution of the supplies. If an attack gets this far it’s too late to stop any damage from happening.

"Downstream Supply Chain": The data, libraries and software and services sold or otherwise distributed to the next level of customers.

It gets confusing because the parts of an IT supply chain look the same at most points. Code doesn’t get refined from ore into shiny ingots, and often it looks almost the same coming out of the manufacturing process as it does coming in to the manufacturing process. And logistics isn’t often a requirement of a computer science degree, so the distinction in parts is not alway intuitive.

The important things to note of these definitions is that they set clear enforcement boundaries and clear incident response scopes. Supply chain attacks should be stopped at the end of the upstream supply chain. This is the most effective place to stop them and has the smallest incident response scope to resolve when attacks are detected here.

Yes you could stop a malicious dependency at a release gate in your CI/CD pipeline. But that’s a bit too late if the developer workstation was the target all along right?

So how do we prevent attacks from reaching our manufacturing process? Well sandboxes are one way to do it, at least for full applications. Install and run them while looking for strange behavior. Maybe run copies of software we provide to our downstream in the sandbox as well with newer dependencies to see if the behavior has changed in unexpected ways? I haven’t seen systems like this, but we know something similar exists not the least because Sunspot had specific code for hiding from sandboxes and detonating in a delayed manner.

One way to do it would be with binary analysis, basically decompiling binaries, to scrutinize new behaviors seems like a possible way to catch upstream supply chain attacks like this. But what vendor is going to be ok with their customers running something like Ghidra on every release of their proprietary software and comparing behaviours?

There is a lot to think about and consider about high tech upstream supply chains. One thing is certain though, they are generally not secure and attacks against them are trending up.

Next >>